Providing crowdsourced answers to information needs presented by search engine and social networking application users

ABSTRACT

Technologies pertaining to generating crowd-sourced answers are described herein. A text string is received, and the text string is parsed to determine if the text string represents an information need that is desirably answered by a collective of crowd workers. When it is determined that the information need is desirably answered by the collective of crowd workers, a query or question that represents the information need is provided to a first plurality of crowd workers, who generate proposed answers for the information need. The proposed answers are provided to a second plurality of crowd workers, who vote on the proposed answers. An answer to the information need is output based upon responses of the crowd workers.

BACKGROUND

Search engines are continuously being adapted to provide relevantinformation to users responsive to receipt of a query. For example, asearch engine results page (SERP) displayed by a conventional searchengine to an issuer of a query includes more information than a list ofweb page titles and snippets retrieved therefrom. For popular topics,such as, weather, movies, and definitions, some search engines haveadded custom interfaces with direct results; for instance, a searchengine can provide the answer of “77 degrees, partly cloudy” to a userwho issues the query “weather in Los Angeles,” wherein such answer isdisplayed inline with web page titles and corresponding snippets. Thesetypes of answers that can be provided to users of a search engine areknown as direct answers, and allow searchers to satisfy an informationneed without having to click through to a web page. Direct answers havea measurable impact on user behavior with respect to SERPs, andoftentimes a user will repeatedly seek direct answers of certain typesonce such user realizes that the search engine can provide the directanswers.

Further, some people have turned to social networking applications toobtain answers to respective information needs. A user of a socialnetworking application can submit a question to a public or private feedin hopes that someone (e.g., a contact of the user) will provide ananswer to the question. Oftentimes, however, the issuer of the questionreceives little or no feedback, causing the information need of the userto remain unsatisfied.

SUMMARY

The following is a brief summary of subject matter that is described ingreater detail herein. This summary is not intended to be limiting as tothe scope of the claims.

Described herein are various technologies pertaining to employing acollective of crowd workers to provide answers to information needs ofusers of a search engine and/or social networking application. Withrespect to a search engine, an exemplary manner in which crowds can beused is to identify portions of Web pages which are likely to includeanswers to information needs (or to directly provide answers to queriesset forth by users of the search engine). For instance, search logs ofthe search engine can be analyzed to identify certain web pages(referred to as candidate web pages) that are believed to includeinformation that satisfies information needs of several users of thesearch engine. In an exemplary embodiment, candidate web pages can beidentified by analyzing end user behavior with respect to queriessubmitted by users, URLs presented to such users, and user interactionwith the URLs. For instance, a web page selected from SERPs that isoften a destination web page (users do not return to respective SERPsand select other search results) can be labeled as a candidate web page.

The candidate web page and queries issued by users that selected thecandidate web page can be transmitted to a first plurality of computingdevices respectively operated by a first plurality of crowd workers in acollective of crowd workers. For instance, the crowd workers in thecollective of crowd workers can be paid workers that are providedcertain monies responsive to completing a specified task. The firstplurality of crowd workers can also be provided with first instructionsfor completing a task, wherein the task is to review content of thecandidate web page and extract (e.g., select, highlight, . . . ) aportion therein that a respective crowd worker believes best answers theinformation need represented by the candidate web page and associatedqueries. Thus, each crowd worker in the first plurality of crowd workersextracts a respective portion of the candidate web page believed to bestanswer the information need, and submits such portion. Portions selectedby crowd workers in the first plurality of crowd workers are transmittedto a second plurality of computing devices operated by a respectivesecond plurality of workers, wherein crowd workers in the secondplurality of crowd workers vote on which portion is believed to be thebest portion for answering the information need. Optionally, the secondplurality of crowd workers can receive other options that may answer theinformation need, such as algorithmically generated answers, such thatthe second plurality of crowd workers has the option to select an optiontransmitted by the first plurality of crowd workers or some otheroption.

The portion of the candidate web page receiving the most votes fromcrowd workers in the second plurality of crowd workers is selected asbeing a candidate answer, and the candidate answer is optionallytransmitted to a third plurality of computing devices operated by arespective third plurality of crowd workers. Instructions are alsotransmitted to the third plurality of computing devices, where theinstructions instruct each crowd worker in the third plurality of crowdworkers to proofread and/or edit the candidate answer to improvereadability. Accordingly, each crowd worker in the third plurality ofcrowd workers may submit a proposed final answer (subsequent toproofreading/editing the candidate answer), and the proposed finalanswers are submitted by the third plurality of crowd workers.

Thereafter, the proposed final answers are transmitted to a fourthplurality of computing devices operated by a respective fourth pluralityof crowd workers. Each crowd worker in the fourth plurality of crowdworkers is instructed to identify which proposed final answer bestanswers the information need represented by the candidate web page andcorresponding queries. The proposed final answer receiving the mostvotes as being the best at answering the information need is selected asthe final answer for the information need. This semi-automated pipelineensures that the final answer for the information need is of highquality and readily consumable by end users of the search engine. Afterthe final answer has been generated, for example, when a user issues aquery that causes the search engine to present the candidate web pagerelatively highly in a SERP, or when a user issues a query that isequivalent to or clustered with queries previously found to beassociated with the candidate web page, the final answer can bepresented to the user inline with conventional search results.

In another exemplary embodiment, the search engine can be adapted toprovide an answer to an information need using a collective of crowdworkers in real-time or near real-time. In such an embodiment, when auser issues a query to the search engine, the query can be analyzed toascertain if it represents an information need that is desirablyanswered by a collective of crowd workers. For instance, the issuer ofthe query may manually indicate that it is desirable that the collectiveof crowd workers provide an answer to the query. In another example, thequery can be analyzed to understand semantic meaning thereof, and thequery can be automatically identified as representing an informationneed that is desirably answered by the collective of crowd workers. Inyet another example, features of the query can be analyzed, and it canbe ascertained that the query represents an information need that istime-critical in nature (e.g., “put out a grease fire”). If it isdetermined that the query is desirably answered by the collective ofcrowd workers, the query (and optionally other information pertaining tocontext of the query) is transmitted to a first plurality of computingdevices operated by a respective first plurality of crowd workers in thecollective of crowd workers with first instructions, wherein the firstinstructions instruct that each crowd worker in the first plurality ofcrowd workers provide a proposed answer to the query. Each crowd workerin the first plurality of crowd workers may then submit a respectiveproposed answer to the query. Each of these proposed answers istransmitted to a second plurality of computing devices operated by arespective second plurality crowd workers in the collective of crowdworkers together with second instructions that instruct crowd workers inthe second plurality of crowd workers to indicate which proposed answeris the best answer to the information need represented by the query.Each crowd worker in the second plurality of crowd workers then votes onwhich of the proposed answers best answers the information need. Theproposed answer receiving the most votes from crowd workers in thesecond plurality of crowd workers may then be provided as a final answerto the user as a portion of a SERP and/or as an alternative to a SERP.As the final answer will be delayed relative to conventional searchresults provided to the issuer of the query, the search results page canbe updated when the final answer is received, or the final answer can betransmitted to the issuer of the query through some other communicationsmedium (e.g., email, instant message, social network message, textmessage, . . . ).

With respect to a social networking application, it has been observedthat users of such applications often submit questions by way of apublic page or a public feed. Such publicly available information can beanalyzed to recognize questions that represent information needsdesirably answered by a collective of crowd workers. An exemplaryquestion that represents an information need that is desirably answeredusing crowd workers can be a question that requests a subjective opinionor a particular fact. Pursuant to an example, a message (sometimesreferred to as an update, a post, or the like) set forth by a user ofthe social networking application can be analyzed to identify if suchmessage includes a question that is desirably answered by the collectiveof crowd workers. For instance, features of the message can be analyzedto ascertain if punctuation is in accordance with a question, if themessage includes certain words known to be question words (e.g., “how”,“where”, “what”, . . . ), if hash tags are included in the message,etc., and the message can be identified as including a question that isdesirably answered by the collective of crowd workers based at least inpart upon such features. Responsive to determining that the messageincludes a question that is desirably answered by the collective ofcrowd workers, the question can be transmitted to a first plurality ofcomputing devices operated by a first plurality of crowd workers in thecollective of crowd workers. Additional information may also be providedto the first plurality of crowd workers, including, but not limited to,candidate web pages that may include an answer to the question, otheranswers to the question submitted by users of the social networkingapplication, etc.

Each crowd worker in the first plurality of crowd workers is instructedto generate a respective proposed answer to the question in the message.The proposed answers can be transmitted to a second plurality ofcomputing devices operated by a respective second plurality of crowdworkers in the collective of crowd workers, and each crowd worker in thesecond plurality of crowd workers votes on which of the proposed answersis the best answer. The proposed answer voted as being the best answercan be submitted to the poster of the message as final answer (e.g., asa private message, posted to the public feed, . . . ).

The above summary presents a simplified summary in order to provide abasic understanding of some aspects of the systems and/or methodsdiscussed herein. This summary is not an extensive overview of thesystems and/or methods discussed herein. It is not intended to identifykey/critical elements or to delineate the scope of such systems and/ormethods. Its sole purpose is to present some concepts in a simplifiedform as a prelude to the more detailed description that is presentedlater.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an exemplary system thatfacilitates presenting, to a user of a search engine or socialnetworking application, a crowd-sourced answer to an information need ofthe user.

FIG. 2 is a functional block diagram of an exemplary system thatfacilitates providing a crowd-sourced answer to a question submitted bya user of a social networking application.

FIG. 3 is a functional block diagram of an exemplary system thatfacilitates generating crowd-sourced answers to information needs ofusers identified through analysis of a search log.

FIG. 4 is a functional block diagram of an exemplary system thatfacilitates providing to a user of a search engine a crowd-sourcedanswer to an information need of the user.

FIG. 5 is a functional block diagram of an exemplary system thatfacilitates providing to a user of a search engine, in real-time or nearreal-time, a crowd-sourced answer to an information need of the user.

FIG. 6 is an exemplary graphical user interface that includes acrowd-sourced answer to a question posted on a public feed of a socialnetworking application.

FIG. 7 is a graphical user interface of an exemplary search engineresults page that includes a crowd-sourced answer.

FIG. 8 is a flow diagram that illustrates an exemplary methodology foroutputting a crowd-sourced answer to an information need of a user.

FIG. 9 is a flow diagram that illustrates an exemplary methodology foroutputting a crowd-sourced answer to an information need expressed by auser in a message posted using a social networking application.

FIGS. 10 and 11 illustrate an exemplary methodology for generatingcrowd-sourced answers to information needs of users of a search engine.

FIG. 12 is a flow diagram that illustrates an exemplary methodology foroutputting a search engine results page that includes a crowd-sourcedanswer.

FIG. 13 is a flow diagram that illustrates an exemplary methodology forgenerating a crowd-sourced answer in response to receipt of a query at asearch engine.

FIG. 14 is an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to providing crowdsourced answers toinformation needs of users will now be described with reference to thedrawings, where like reference numerals represent like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of one or more aspects. It may be evident, however, thatsuch aspect(s) may be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate describing one or more aspects. Further, itis to be understood that functionality that is described as beingcarried out by certain system components may be performed by multiplecomponents. Similarly, for instance, a component may be configured toperform functionality that is described as being carried out by multiplecomponents.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Further, as used herein, the terms “component” and “system” are intendedto encompass computer-readable data storage that is configured withcomputer-executable instructions that cause certain functionality to beperformed when executed by a processor. The computer-executableinstructions may include a routine, a function, or the like. It is alsoto be understood that a component or system may be localized on a singledevice or distributed across several devices. Additionally, the term“exemplary” is intended to mean serving as an illustration or example ofsomething, and is not intended to indicate a preference.

With reference now to FIG. 1, an exemplary system 100 that facilitatesprovision of crowd-sourced answers to users of at least one of a searchengine or social media application responsive to such users expressinginformation needs that are desirably answered by a collective of crowdworkers is illustrated. A user 102 operates a computing device 104 toaccess at least one of a search engine or social networking application.The computing device 104 can be any suitable computing device, includinga desktop computer, a laptop computer, a mobile telephone, a tabletcomputing device, a portable media player, etc. In a non-limitingexample, the user 102 can operate the computing device 104 to cause aweb browser to be initiated and can direct the web browser to a URLcorresponding to the at least one of the search engine or the socialnetworking application. In other embodiments, the computing device 104may have standalone applications installed thereon for the at least oneof the search engine or the social networking application.

The user 102 operates the computing device 104 to cause express aninformation need to the search engine or the social networkingapplication. Generally, with respect to the search engine, theinformation need is expressed in the form of a query submitted by theuser 102. With respect to the social networking application, theinformation need can be expressed in the form of a question included ina message posted on a public page or feed by the user 102.

The system 100 includes an answer system 106 that is employed to providea crowd-sourced answer to the information need expressed by the user 102operating the computing device 104. In an example, the answer system 106may be included in a search engine. In another example, the answersystem 106 can monitor messages posted by users of the social networkingapplication. In yet another example, the answer system 106 can beincluded in the social networking application.

The answer system 106 includes a classifier component 108 that receivesthe expression of the information need (the query or message), anddetermines if the information need is one that is desirably answered bya collective of crowd workers 110. The classifier component 108 canutilize various techniques when if the information need is desirablyanswered by the collective of crowd workers 110 (rather than aninformation need that is not desirably answered by the collective ofcrowd workers 110). For instance, the user 102 of the computing device104 can indicate, when setting forth the query or message, that theinformation need represented thereby is desirably answered by thecollective of crowd workers 110. The classifier component 108 canreceive such indication and can classify the information needaccordingly.

In another example, the classifier component 108 can parse the text ofthe query or message to ascertain semantic meaning of the query orquestion, and determine if the corresponding information need isdesirably answered by the collective of crowd workers 110. For instance,the classifier component 108 can ascertain that a question submitted bythe user 102 is rhetorical in nature, in which it would be undesirablefor the collective of crowd workers 110 to provide an answer to theinformation need represented by such question.

In yet another example, the classifier component 108 can analyzefeatures corresponding to the query or message. Exemplary featuresinclude whether the query or message includes a question mark, whetherthe query or message includes at least one word from a predefined listof words (such as “who,” “what,” “when,” “where,” “why,” “how,” and“which”), whether the query or message includes a certainnon-alphanumerical character, such as the “#” symbol, amongst others.For example, users of a relatively popular social networking applicationemploy hash tags to label messages. The classifier component 108 canclassify the information need represented by the message as being onethat is desirably answered by the collective of crowd workers 110 basedat least in part the message including a hash tag.

In still yet another example, the classifier component 108 can beconfigured to transmit the query or message to a crowd worker in thecollective of crowd workers 110, and the crowd worker can indicatewhether or not the information need represented by the query or messageis desirably answered by the collective of crowd workers 110. Theclassifier component 108 can determine whether such information need isdesirably answered by the collective of crowd workers 110 based at leastin part upon such indication.

Still further, the classifier component 108 can be configured to performa semantic analysis on the text string to ascertain a subject/topic ofthe information need. For instance, topics can be identified a priori,and the classifier component 108 can be configured to perform thesemantic analysis on the text string to classify the text string asbelonging to a particular topic, a set of topics, or no defined topic.

The collective of crowd workers 110 comprises a first plurality of crowdworkers 112-114 that operate a respective first plurality of computingdevices 116-118. The collective of crowd workers 110 further includes asecond plurality of crowd workers 120-122 that operate a respectivesecond plurality of computing devices 124-126. In an example, a numberof crowd workers in the first plurality of crowd workers 112-114 can bebetween three and five workers. Additionally, a number of crowd workersin the second plurality of crowd workers 120-122 can be between threeand five workers. Crowd workers in the collective of crowd workers 110can be geographically dispersed, and may or may not be paid crowdworkers. For instance, the crowd worker 112 in the first plurality ofcrowd workers can reside in a first country, while the crowd worker 114in the first plurality of crowd workers can reside in a second country.Further, the collective of crowd workers 110 can be provided by a paidservice. In other examples, the collective of crowd workers 110 can becomposed of volunteers, contacts of the user 102, etc.

The answer system 106 is employed in connection with generating ananswer to the information need expressed by the user 102 operating thecomputing device 104 through utilization of the collective of crowdworkers 110. The answer system 106 includes an instruction transmittercomponent 128 that, responsive to the classifier component 108determining that the information need of the user 108 is desirablyanswered by the collective of crowd workers 110, transmits theexpression of the information need (the query or message), firstinstructions, and optionally other information to the first plurality ofcomputing devices 116-118 operated by the respective first plurality ofcrowd workers 112-114. The first instructions instruct each crowd workerin the first plurality of crowd workers 112-114 to perform a same task.In an exemplary embodiment, the task provided to the first plurality ofcrowd workers 112-114 can be to generate a proposed answer to theinformation need of the user 102, as expressed in the query or message.In an exemplary embodiment, the instruction transmitter component 128can identify the first plurality of crowd workers 112-114 from amongst alarger set of crowd workers based upon identified “expertise” of suchcrowd workers and a topic of the information need identified by theclassifier component 108. Thus, if the classifier component 108 hasidentified the information need as belonging to a certain topic (e.g.,medicine related), then the instruction transmitter component 128 canidentify the first plurality of crowd workers 112-114 as having someexpertise in such topic.

The first plurality of crowd workers 112-114 operate the respectivefirst plurality of computing devices 116-118 to generate proposedanswers to the information need of the user. As noted above, the firstplurality of crowd workers 112-114 can be provided with supplementalinformation that can assist them in generating proposed answers to theinformation need. Such supplemental information can include a resourcethat may be of assistance to crowd workers in answering the informationneed. An exemplary resource may include, but is not limited toincluding, at least one candidate web page that is believed to includean answer to the information need of the user 102, messages posted inresponse to the message set forth by the user 102, algorithmicallygenerated answers, contextual information about the user 102 (providedwith consent of the user 102), etc. Each crowd worker in the firstplurality of crowd workers 112-114 employs her computing device in thefirst plurality of computing devices 116-118 to submit a respectiveproposed answer to the answer system 106.

The answer system comprises a response receiver component 130 thatreceives a response to the task (a proposed answer) from each computingdevice in the first plurality of computing devices 116-118. Responsiveto the response receiver component 130 receiving the proposed answers tothe information need from the first plurality of computing devices116-118, the instruction transmitter component 128 can transmit theproposed answers and second instructions to the second plurality ofcomputing devices 124-126 operated by the respective second plurality ofcrowd workers 120-122. Optionally, the instruction transmitter component128 can transmit proposed answers generated from a source other than thefirst plurality of crowd workers 112-114, such as algorithmicallygenerated answers. Each crowd worker in the second plurality of crowdworkers 120-122 therefore receives each proposed answer submitted bycrowd workers in the first plurality of crowd workers 112-114 (andoptionally other proposed answers). The second instructions transmittedby the instruction transmitter component 128 instruct crowd workers inthe second plurality of crowd workers 120-122 to indicate which proposedanswer from amongst the proposed answers best answers the informationneed of the user 102. Thus, the second plurality of crowd workers120-122 can employ the respective second plurality of computing devices124-126 to submit votes to the answer system 106 as to which proposedanswer best answers the information need of the user 102.

The response receiver component 130 receives the votes from the secondplurality of computing devices 124-126. The answer system 106 furtherincludes an output component 132 that, responsive to the receivercomponent 130 receiving the votes, tabulates the votes and selects theproposed answer received the most votes as a final answer to theinformation need of the user 102. The output component 132 transmits thefinal answer to the computing device 104 operated by the user 102, suchthat the final answer can be displayed to the user 102. As will bedescribed in greater detail below, if the answer system 106 is employedin connection with a search engine, the output component 132 can causethe final answer to be displayed inline with conventional web-basedsearch results generated by the search engine. If the answer system 106is employed in connection with a social networking application, theoutput component 132 can cause the final answer to be included in apublic feed of the social networking application (in correspondence withthe message posted by the user 102) and/or posted on a public page.

While the answer system 106 has been described as being employed inconnection with a search engine or social networking application, inother embodiments the answer system 106 can be employed in connectionwith other applications. For instance, the user 102 can request that theanswer system 106 analyze instant messages generated by the user 102 byway of an instant messaging application and provide crowd-sourcedanswers to information needs expressed in such instant messages.Similarly, the user 102 can request that the answer system 106 analyzeemails generated by the user 102 by way of an email application andprovide crowd-sourced answers to information needs expressed in suchinstant messages. It is thus to be understood that the user 102 mayrequest that the answer system 106 be employed with respect to any textgenerated by such user.

Now referring to FIG. 2, an exemplary system 200 that facilitatesprovision of a crowd-sourced answer to a question included in a messageposted by a user of a social networking application is illustrated. Inthe exemplary system 200, the user 102 operates the computing device 104to access a social networking application 202. In an exemplaryembodiment, the social networking application 202 can be a messagebroadcasting application, where users of the social networkingapplication 202 can broadcast relatively short messages to a public feedor to other users of the social networking application that subscribe tomessages posted by the user 102. A public feed of the social networkingapplication 202 comprises a series of messages posted by users of thesocial networking application 202 that are available for consumption bythe public. Aspects described with respect the system 200, however, arenot limited to this type of social networking application. For example,other social networking applications allow users to post messages(status updates) on public pages. Thus, if the individual chooses topost messages on a public page, then other people who are not contactsof the user 102 can review messages posted by the user 102 on suchpublic page.

For purposes of explanation, the social networking application 202 willbe described as being a message broadcasting application, although itwill be readily apparent that other types of social networkingapplications are contemplated. As noted above, the user 102 employs thecomputing device 104 to post a message by way of the social networkingapplication 202 to a public feed. The answer system 106 monitors thepublic feed for messages that include questions that are desirablyanswered by the collective of crowd workers 110. While shown as beingseparate from the social networking application 202, it is to beunderstood that in some embodiments, the answer system 106 may beincluded in the social networking application 202. The answer system 106comprises an analyzer component 204 that monitors the public feed 202for messages that may include questions. For instance, a message postedby the user 102 may include the question “which is better, peanut butteror jelly?” The analyzer component 204 can parse text of the message anddetermine that the message includes words typically associated with aquestion, such as, “who,”0 “what,” “when,” “where,” “why,” “how,”“which,” and/or the like. Further, the analyzer component 204 can lookfor punctuation, such as a question mark, to determine that the messageincludes a question.

The classifier component 108 is in communication with the analyzercomponent 204, and the classifier component 108 can receive anindication from the analyzer component 204 that a message has beenposted by the user 102 that includes a question. The classifiercomponent 108 may then further analyze the question to ascertain if itis desirably answered by the collective of crowd workers 110. With morespecificity, the classifier component 108 can analyze various featuresof the message to determine if the question therein is desirablyanswered by the collective of crowd workers 110. In an example, theclassifier component 108 can analyze the question to determine if it istime critical in nature. This can be indicated, for instance, by thequestion being set forth in all capital letters, by the inclusion of anexclamation point together with a question mark in the question, etc.Moreover, the classifier component 108 can be configured to ignore thequestion if the message posted by the user 102 is a reposting of amessage generated by another user. Likewise, the classifier component108 can be configured to ignore the question if the message comprisingthe question includes a URL.

In still yet another example, the classifier component 108 can determinethat the question is desirably answered by the collective of crowdworkers 110 based at least in part upon a non-alphanumerical characterin the message that is positioned in correspondence with the question.For instance, currently, many users of a particular type of socialnetworking application include hash tags in messages (e.g., #help),where a hash tag is used by an author of a message to label suchmessage. Thus, the classifier component 108 can be configured toclassify the question as being one which is desirably answered by thecollective of crowd workers 110 based upon the inclusion of a hash tagin the message that includes the question. The classifier component 108,as noted above, can also use other automated filtering methods, such asperforming a semantic analysis on the question to ascertain if thequestion is rhetorical in nature, is requesting a subjective opinion, isrequesting a fact or listing of facts, etc.

Subsequent to the classifier component 108 utilizing such filteringtechniques, if it is determined that the question is one that maydesirably be answered by the collective of crowd workers 110, theclassifier component 108 can optionally transmit the question to acomputing device 206 of a crowd worker 208 in the collective of crowdworkers 110, wherein the crowd worker 208 is instructed to make a finaldetermination as to whether the question is desirably answered by thecollective of crowd workers 110. For instance, if not undertaken by theclassifier component 108, the crowd worker 208 can be asked if thequestion is a rhetorical question or one that requires subjective input.If the question is rhetorical in nature, the crowd worker 208 canindicate as much, and a crowd-sourced answer is not provided to the user102. If the crowd worker 208 indicates that the question requiressubjective input, is requesting a particular fact or facts, isrequesting a list and/or is time critical in nature, the crowd worker208 can cause the computing device 206 to transmit an indication to theanswer system 106 that the question is desirably answered by thecollective of crowd workers 110. The classifier component 108 canreceive such indication, and responsive to receiving the indication, canclassify the question as being desirably answered by the collective ofcrowd workers 110.

Responsive to the classifier component 108 indicating that the questionin the message posted by the user 102 is desirably answered by thecollective of crowd workers 110, the instruction transmitter component128 can transmit the question, first instructions, and (optionally)supplemental information to the first plurality of computing devices116-118 operated by the respective first plurality of crowd workers112-114. The first instructions can instruct crowd workers in the firstplurality of crowd workers 112-114 to set forth an answer to thequestion. The supplemental information (also referred to as at least oneresource) includes information that may be of assistance to crowdworkers in the first plurality of crowd workers 112-114 when formulatingrespective answers to the question. Such supplemental information mayinclude, but is not limited to including, at least one candidate webpage that possibly includes an answer to the question, otherquestions/queries related to the question, responses to the messageposted by other users of the social networking application 202,algorithmically generated answers, publicly available information aboutthe user 102 (e.g., information in or relating to a user profile for thesocial networking application 202), information about the user 102voluntarily provided by the user, etc.

Utilizing the first plurality of computing devices 116-118, crowdworkers in the first plurality of crowd workers 112-114 submitrespective proposed answers to the question to the answer system 106.The response receiver component 130 receives the proposed answerssubmitted by the crowd workers in the first plurality of crowd workers112-114. Responsive to the response receiver component 130 receiving theproposed answers, the instruction transmitter component 128 transmitsthe proposed answers, the question, second instructions, and(optionally) supplemental information to the second plurality ofcomputing devices 124-126 operated by the respective second plurality ofcrowd workers 120-122. Additionally, and optionally, the instructiontransmitter component 128 can transmit proposed answers generated bysources other than the first plurality of crowd workers 112-114, such asalgorithmically generated proposed answers. The second instructionsinstruct the crowd workers in the second plurality of crowd workers120-122 to submit an indication (vote) as to which of the proposedanswers best answers the question proffered by the user 102.Accordingly, each crowd worker in the second plurality of crowd workers120-122 receives each proposed answer submitted by crowd workers in thefirst plurality of crowd workers 112-114 (and optionally proposedanswers from other sources) and votes on which of the proposed answersbest answers the question set forth by the user 102. Using the secondplurality of computing devices 124-126, the respective second pluralityof crowd workers 120-122 submits votes to the answer system 106.

The response receiver component 130 receives the votes, and responsiveto the response receiver component 130 receiving the votes, the outputcomponent 132 tabulates the votes and selects the proposed answer thatreceives the highest number of votes as being a final answer to thequestion. If two or more proposed answers have the same number of votes(and that number is the highest number of votes), the output component132 may randomly select one of such answers or may select both of suchanswers. The output component 132 may output a message for posting tothe public feed, wherein the message includes the final answer.Additionally, the message output by the output component 132 can bepositioned in the public feed to indicate that it is a response to themessage set forth by the user 102. In other examples, the outputcomponent 132 can cause an instant message, text message, e-mailmessage, or the like, to be transmitted to an account of the user 102.

The answer system 106 further optionally includes a quality component210 that is configured to ensure that crowd workers in the collective ofcrowd workers 110 are adequately performing tasks assigned thereto. Forexample, the quality component 210 can, from time to time, provide aquestion to crowd workers in the collective of crowd workers, where thequestion has been labeled with a ground truth. The quality component 210can create ground truth tasks for which an answer of a worker must meetcertain standards (e.g., a proposed answer must include a word or phraseand/or must not include a certain word or phrase). The quality component210 may then grade individual crowd workers based upon responses tothese standardized tasks. If a crowd worker is deemed to perform poorlyover time (or very poorly a single time), then the quality component 210can cause the instruction transmitter component 128 to fail to sendfurther tasks to such crowd worker. The quality component 210 may alsoutilize user feedback. For instance, if the user 102 indicates that theanswer to the question is of poor quality, the quality component 210 canidentify which crowd workers in the collective of crowd workers 110contributed to the answer and grade such crowd workers accordingly. If agrade of a crowd worker over time falls below some threshold, then theinstruction transmitter component 128 can be configured to fail totransmit subsequent tasks to such crowd worker.

The answer system 106 may also optionally include an accountingcomponent 212 that monitors tasks completed by crowd workers in thecollective of crowd workers 110. For example, as noted above, the crowdworkers in the collective of crowd workers 110 may be paid crowdworkers, such that they are provided a particular fee for performing acertain task. The accounting component 212 can keep an accounting oftasks performed by individual crowd workers, such that the crowd workerscan be appropriately paid.

Turning now to FIG. 3, an exemplary system 300 that facilitatesutilizing the collective of crowd workers 110 to generate answers forinformation needs of users of a search engine (identified throughanalysis of search logs) is illustrated. The system 300 includes a datarepository 302 that comprises a search log 304. The search log 304includes a plurality of search sessions of users of the search engine. Asearch session includes a search query submitted to a search engine andsubsequent action undertaken by the issuer of the query (selection ofURLs on a SERP, query reformulation, etc.).

The answer system 106 comprises a candidate identifier component 306that analyzes the search log 304 to identify web pages that are believedto include answers to information needs of users. These web pages arereferred to as candidate web pages. In connection with identifyingcandidate web pages, the candidate identifier component 306 extractssearch trails from the search log 304. A search trail is a browsing pathbeginning with a query submitted by a user and terminating with asession timeout of 30 minutes. The candidate identifier component 306groups all search trails on a first clicked search result (URL) from acorresponding SERP. Accordingly, the candidate identifier component 306can identify a set of queries that led to a particular URL and a set oftrails that describe what issuers of the queries did subsequent toclicking through to the URL (e.g., return to the SERP and select anotherURL, reformulate the query, remain on the selected URL, . . . ). Thus,for example, the candidate identifier component 306 can identify URLsthat are selected by users some threshold number of times when includedon a SERP, wherein when the URLs are selected the respective usersterminate their respective search sessions. These identified URLs(candidate web pages) and queries submitted by search engine users whenselecting the URLs can be retained, wherein a candidate web page andcorresponding queries are referred to as candidate information needs.

A filter component 308 is in communication with the candidate identifiercomponent 306. The filter component 308 identifies information needsidentified by the candidate identifier component 306 that are intendedfor fact finding. Some information needs are too complex to answer,while others have underspecified queries where the information needrepresented by such queries may be unclear. The filter component 308 canutilize any suitable filtering techniques to identify which candidateinformation needs are desirably satisfied by answers set forth by thecollective of crowd workers 110.

With more specificity, the filter component 308 can use search trails toidentify web pages where users quickly end search sessions. Forinstance, it can be assumed that after submitting query to a searchengine and reviewing at least one web page identified in thecorresponding SERP, users typically end up at web pages that includeinformation that addresses their respective information needs. If a userceases browsing after they reach a web page, such page likely includesinformation that satisfies the information need of the user. If the userreaches a web page and thereafter continues browsing or searching, onthe other hand, the web page may not succinctly satisfy the informationneed of the user. For example, many queries are navigational in nature,in that searchers click on a particular URL in the results, then oftenkeep browsing in the page corresponding to the URL (e.g., by clicking ona link in the page). Other information needs, such as buying a new car,are complex and persist across multiple sessions, so users will oftenaccess several pages in a SERP. For many other queries, however, theuser will issue a query, click through to a page shown in the SERP,locate the information that is desired, and end the search session.

Accordingly, the filter component 308 can filter candidate web pages(and thus candidate information needs) utilizing a metric that can bereferred to as destination probability. The destination probability fora web page is an observed probability that a searcher will end hersession at that web page after clicking through to the page from thesearch results page. For example, a step immediately after the userissuing a query can be a click on web page shown in the SERP. If a highpercentage of trails end after such click (e.g., if the trail length istwo), the destination probability will be high. If most trails, instead,include actions that return to the SERP or browse to other URLs, thedestination probability will be low. In other words, the destinationprobability for a URL is the observed probability that a click to theURL from the SERP is the last action in the search trail. Candidate webpages with destination probability above a predefined threshold can beidentified by the filter component 308 as corresponding to aninformation need that may be desirably answered by the collective ofcrowd workers 110. For instance, the filter component 308 can filter outany candidate web pages that have destination probability of less than0.3.

The filter component 308 can also filter information needs based uponinclusion of words that typically pertain to a question in a query. Withmore particularity, destination probability identifies web pages wheresearchers appear to be finding immediate answers for their informationneeds. It can be very difficult to infer the fact-finding intent,however, from queries that are only two or three words long. Forinstance, an answer for the query “dissolvable stitches” would bevaluable if the searcher wanted to learn how long the stitches take todissolve, but would not be valuable if the searcher wanted to learn thehistory of dissolvable stitches.

To avoid such problem, the filter component 308 can make use of queriesthat include question-type words. Such words are useful, because theytend to be expressed in natural language, are longer than typicalqueries, and are more explicit (e.g., “how long do dissolvable stitcheslast”). Such properties make the information need relatively easy tounderstand. Use of question words also tends to indicate fact-findingintent. It can be assumed that question-word queries often overlapsignificantly with unspecified information needs from other queries. Forexample, different users issuing the queries “where is 732 area code”and “732 area code” may have similar information needs. The filtercomponent 308 can remove candidate web pages that have fewer than somethreshold percentage of their clicks (e.g., one percent) fromquestion-word queries. Question words that can be employed by the filtercomponent 308 can include “how,” “why,” “when,” “where,” “why,” “who,”“which,” and the like.

The filter component 308 can also be configured to filter candidate webpages based upon answer type. While question words are useful foridentifying candidate information needs, neither they, nor other typesof behavioral log data, assist in understanding whether a concise answercould address an information need of a user. Having understanding of anexpected length of an answer may be important, since crowd may extracttoo much text in order to guarantee that the correct information iscaptured (and, thus, guarantee that the crowd worker will be paid).Answer candidates can be categorized into different types. Short answersare answers that include very little text (less than 100 characters).List answers are those that include a relatively small set ofdirections. For example, “to change your password, first click a certainhyperlink, then click a button, and thereafter click the ‘changepassword button’”. Summary answers are those that synthesize largeamounts of content.

Responsive to the filter component 308 identifying a set of candidateinformation needs, the instruction transmitter component 128 cantransmit a candidate information need (which includes a candidate webpage and queries corresponding to the candidate web page identified fromthe search log 304) and first instructions to the first computingdevices 116-118 operated by the respective first plurality of crowdworkers 112-114. The first instructions can request that each crowdworker in the first plurality of crowd workers 112-114 extract as littletext as possible from the candidate web page (using the correspondingqueries as a guide), wherein the extracted text is believed by the crowdworker to best answer the candidate information need. Accordingly, eachcrowd worker in the first plurality of crowd workers 112-114 can employa respective computing device in the first plurality of computingdevices 124-126 to submit a portion the candidate web page believed tobest satisfy the candidate information need (represented by the at leastone query and the candidate web page).

The response receiver component 130 receives the submitted portions ofthe candidate web page from crowd workers in the first plurality ofcrowd workers 112-114. The instruction transmitter component 128, inresponse to the response receiver component 130 receiving the portionsof the candidate web page submitted by crowd workers in the firstplurality of crowd workers 112 through 114, can transmit the portions ofthe candidate web page, the candidate web page, and second instructionsto the second plurality of computing devices 124-126 operated by therespective second plurality of crowd workers 120-122. The secondinstructions instruct each crowd worker in the second plurality of crowdworkers 120-122 to vote on which portion of the portions of thecandidate web page identified by crowd workers in the first plurality ofcrowd workers 112-114 best answers the information need. Each crowdworker in the second plurality of crowd workers 120-122 uses arespective computing device in the second plurality of computing devices124-126 to submit a vote to the answer system 106 as to which portionbest answers the information need.

The response receiver component 130 receives the votes submitted by thesecond plurality of crowd workers 120-122, tabulates the votes, andselects the portion of the candidate web page receiving the highestnumber of votes. In an exemplary embodiment, the output component 132can output the portion of the web receiving the highest number of votesas the best answer to the information need, and such portion can beretained in a search engine repository 310, indexed by, for instance, aweb page from which the portion was extracted and/or queriescorresponding to the web page. As will be described below, such searchengine repository 310 can subsequently be accessed to provide searcherswith crowd-sourced answers.

Optionally, responsive to the response receiver component 130 receivingthe aforementioned votes, the instruction transmitter component 128 cantransmit the portion receiving the most votes, the queries correspondingto such portion, and third instructions to a third plurality ofcomputing devices 312-314 operated by a third plurality of crowd workers316-318. The third instructions instruct the third plurality of crowdworkers 316-318 to propose edits to the portion of the candidate webpage, wherein the edits are to summarize such portion, improvereadability of such portion, or cause such portion to conform topredefined rules set forth by the search engine (e.g., no longer thanfifteen words, include correct punctuation, . . . ). Each crowd workerin the third plurality of crowd workers 316-318 can independently editthe aforementioned portion of the candidate web page, and can use arespective computing device in the third plurality of computing devices312-314 to submit edited portions to the answer system 106. The responsereceiver component 130 receives the edited portions, and responsive tothe response receiver component receiving the edited portions, theinstruction transmitter component 128 can transmit the edited portions,the queries, and fourth instructions to a fourth plurality of computingdevices 320-322 operated by a fourth plurality of crowd workers 324-326.The fourth instructions instruct each crowd worker in the fourthplurality of crowd workers 324-326 to submit a respective vote as towhich edited portion is the best edited portion. The fourth plurality ofcrowd workers 324-326 use the respective fourth plurality of computingdevices 320-322 to submit the votes to the answer system 106. Theresponse receiver component 130 can receive votes submitted by crowdworkers in the fourth plurality of crowd workers 324-326, tabulate thevotes, and select the edited portion receiving the highest number ofvotes as being an answer to the information need.

Responsive to the response receiver component 130 receiving the votesfrom the fourth plurality of computing devices 320-322, the instructiontransmitter component 128 can transmit the answer, the queries, andfifth instructions to a fifth plurality of computing devices 328-330operated by a fifth plurality of crowd workers 332-334. The fifthinstructions can instruct crowd workers in the fifth plurality of crowdworkers 332-334 to assign a short title (e.g., five words or less) tothe answer system 106. Each crowd worker in the fifth plurality of crowdworkers 332-334 uses a respective computing device in the fifthplurality of computing devices 328-330 to submit respective short titlesto the answer system 106.

Responsive to the response receiver component 130 receiving proposedshort titles from the fifth plurality of computing devices 328-330, theinstruction transmitter component 128 can transmit such short titles,the answer, and sixth instructions to a sixth plurality of computingdevices 336-338 operated by a respective sixth plurality of crowdworkers 340-342. The sixth instructions can instruct each crowd workerin the sixth plurality of crowd workers 340-342 to submit a respectivevote as to which short title is the best short title. The sixthplurality of crowd workers 340-342 operate the sixth plurality ofcomputing devices 336-338 to submit votes to the answer system 106.

The response receiver component 130 can receive such votes, tabulate thevotes, and select the short title with the greatest number of votes asbeing a title for the answer. The output component 132 can cause theanswer and the short title to be retained in the search enginerepository 310. Using this approach, many answers and short titles canbe generated by the collective of crowd workers 110 for variousinformation needs of users of the search engine identified in the searchlog 304.

As mentioned above, the answer system 106 can optionally include thequality component 210 to ensure that crowd workers in the collective ofcrowd workers 110 are adequately following the instructions. The qualitycomponent 210 can incorporate what is referred to as the “gold standardtechnique”, which requires that crowd workers demonstrate competence byagreeing with answers to pre-authored example questions for a task. Thequality component 210 can silently insert gold standard questions into awork queue of a crowd worker, and crowd workers who fail to answer athreshold number of gold standard questions correctly can be identified,such that future tasks are not assigned to such crowd workers.

Further, the quality component 210 can incorporate aninclusion/exclusion list for gold standard testing with respect to thefirst plurality of crowd workers 112-114. To use an inclusion/exclusionlist in connection with gold standard testing for the first plurality ofcrowd workers 112-114, a crowd worker must extract sections of the pageincluded in the inclusion list and must not extract sections of the pageincluded in the exclusion list.

Now referring to FIG. 4, an exemplary search engine 400 that includes apreviously generated crowd-sourced answer (e.g., generated throughutilization of the system 300) in a SERF responsive to the user 102submitting a query to the search engine 400 by way of the computingdevice 104 is illustrated. The search engine 400 includes a queryreceiver component 402 that receives a query submitted by the user 102of the computing device 104. A search component 404 is in communicationwith the query receiver component 402, and executes a search over asearch engine index 406 based at least in part upon the query. Executionof the search results in retrieval of a ranked list of URLs. The searchcomponent 404 can access the search engine repository 310 responsive toretrieving the ranked list of URLs and ascertain if a URL in the searchengine repository 310 (which has a crowd-sourced answer based at leastin part upon content of a web page corresponding to the URL) is includedin some top threshold number of URLs in the ranked list of URLs. If thesearch results include a URL (e.g., as one of the top three searchresults) that is identified in the search engine repository, the searchcomponent 404 can retrieve the crowd-sourced answer corresponding tosuch URL and include the answer inline with the search results.Additionally, the answer can be highlighted to indicate that the answeris a crowd-sourced answer.

In a specific example, the search component 404 can receive a query,execute a search over the search engine index 406, and generate a rankedlist of search results, wherein a first URL (URL 1) is the second mosthighly ranked search result in the ranked list of search results. Thesearch component 404 can then access the search engine repository 310,which includes a list of URLs that have crowd-sourced answerscorresponding thereto. The search component 404 can determine that thesearch engine repository 310 includes URL 1, and can extract thecorresponding answer (answer 1) from the search engine repository 310and include answer 1 inline with the ranked list of search results.

Now referring to FIG. 5, an exemplary search engine 500 that can providesearch results that include an answer generated by the collective ofcrowd workers 110 in real-time or near real-time is illustrated. Theuser 102 of the computing device 104 submits a query to the searchengine 500. The search engine 500 includes the answer system 106. Thesearch component 404 receives the query and executes a search over thesearch engine index 406 to locate web page that are believed to berelevant to the query set forth by the user 102. The search component404 may then (immediately) output the search results on a SERP to thecomputing device 104 of the user 102.

Additionally, the classifier component 108 of the answer system 106 canreceive the query and determine if the query represents an informationneed that is desirably answered by the collective of crowd workers 110.As noted above, the classifier component 108 can determine if it isdesirable for the collective of crowd workers 110 to provide an answerbased upon various features corresponding to the query. Such featurescan include whether the query is written in the form of a question,whether the classifier component 108 finds the query to be time-criticalin nature, whether the query is believed to be searching for a fact orshort list, etc. If the classifier component 108 determines that thequery does not represent an information need that is desirably answeredby the collective of crowd workers 110, then the user 102 can review theURLs in the SERP to satisfy the information need.

If, however, the classifier component 108 deems that the query submittedby the user 102 represents an information need that is desirablyanswered by the collective of crowd workers 110, then the classifiercomponent 108 can output an indication to the computing device 104 thatthe collective of crowd workers 110 are being employed to generate acrowd-sourced answer. Such indication can be displayed on the SERP,transmitted in the form of an instant message, or other suitable mannerof notifying the user 102 that the collective of crowd workers 110 arebeing employed to generate the crowd-sourced answer.

Responsive to the classifier component 108 deeming that the querysubmitted by the user 102 represents an information need that isdesirably answered by the collective of crowd workers 110, theinstruction transmitter component 128 can transmit the query,corresponding information, and first instructions to the first pluralityof computing devices 116-118 operated by the respective first pluralityof crowd workers 112-114. The corresponding information can includealgorithmically generated answers, context pertaining to the user 102(e.g., information from a public profile of the user 102, informationexplicitly provided by the user 102, at least one web page included inthe SERP, . . . ). The first instructions instruct the first pluralityof crowd workers 112-114 to generate proposed answers for theinformation need of the user 102. Using the first plurality of computingdevices 116-118, the first plurality of crowd workers 112-114 submit theproposed answers to the answer system 106.

The response receiver component 130 receives such proposed answers, andresponsive to the response receiving component 130 receiving theproposed answers, the instruction transmitter component 128 transmitsthe answers, the query, (optionally) the corresponding information, andsecond instructions to the second plurality of computing devices 124-126operated by the respective second plurality of crowd workers 120-122.The second instructions instruct the second plurality of crowd workers120-122 to submit votes to the answer system 106 as to which of theanswers generated by crowd workers in the first plurality of crowdworkers 112-114 best answers the information need of the user 102 (asrepresented by the query). The crowd workers in the second plurality ofcrowd workers 120-122 operate the respective second plurality ofcomputing devices 124-126 to submit the votes to the answer system 106.The response receiver component 130 receives the votes, tabulates thevotes, and selects the answer that receives the highest number of votesas a final answer. The output component 132 can output the final answerto the computing device 104 of the user 102. As mentioned above, theSERP presented to the user 102 can be updated to include the finalanswer, such that the final answer is shown inline with search resultson the SERP previously shown to the user. Additionally or alternatively,the output component 132 can transmit the final answer to an account ofthe user 102, such as an email account, an instant messaging account, asocial networking account, etc. Transmitting the final answer to anaccount of the user 102 may be particularly beneficial when the user 102is, for some reason, in a hurry, and wishes to perform another taskwhile the final answer is being generated by the answer system 106.

Turning now to FIG. 6, an exemplary graphical user interface 600 of apublic feed of a social networking application is illustrated. Thepublic feed includes a message 602 that is posted by a user of thesocial networking application. The message 602 can include a graphicalicon 604 (such as an avatar) that identifies the poster of the message,as well as a text identifier (such as a handle) that further identifiessuch poster. The message 602 also includes a question submitted by theposter of the message 602.

In the example graphical user interface 600, a second message 606 isposted by a contact (e.g., a follower) of the poster of the firstmessage 602. The second message 606 includes a second graphical icon 608that identifies a poster of the second message, a text identifier thatfurther identifies the poster of the second message, as well as, forinstance, an answer to the question included in the first message 602.

The question in the first message 602 and (optionally) the answer in thesecond message 606 can be transmitted to crowd workers in the collectiveof crowd workers 110. The collective of crowd workers 110 can generatean answer, which can be presented to the poster of the first message 602as a portion of the public feed in correspondence with the first message602. For instance, the public feed can include a third message 610 thatcomprises a graphical icon 612 that identifies the collective of crowdworkers 110 (e.g., identifies that an answer in the third message 610 isa crowd-sourced answer) as well as a text identifier that furtheridentifies the collective of crowd workers 110. The third message 610also includes an answer to the question set forth in the first message602.

With reference now to FIG. 7, a graphical user interface 700 of a SERFoutput by a search engine is illustrated. The graphical user interface700 includes a query field 702, wherein a user can set forth a query tothe search engine by way of the query field 702. The graphical userinterface 700 further includes a button 704, wherein, in an example, aquery entered into the query field 702 is transmitted to the searchengine responsive to the user selecting the button 704.

The graphical user interface 700 includes a plurality of search resultspresented by the search engine upon executing a search over a searchengine index based upon the query. The search results include aplurality of web page identifiers 706-710. The search results alsoinclude an answer 712 generated by the collective of crowd workers 110in a manner described above. The answer 712 can be highlighted toindicate that it a crowd-sourced answer (rather than a conventionalsearch result).

FIGS. 8-13 illustrate exemplary methodologies relating to generatingcrowd-sourced answers to information needs of users. While themethodologies are shown and described as being a series of acts that areperformed in a sequence, it is to be understood and appreciated that themethodologies are not limited by the order of the sequence. For example,some acts can occur in a different order than what is described herein.In addition, an act can occur concurrently with another act. Further, insome instances, not all acts may be required to implement a methodologydescribed herein.

Moreover, the acts described herein may be computer-executableinstructions that can be implemented by one or more processors and/orstored on a computer-readable medium or media. The computer-executableinstructions can include a routine, a sub-routine, programs, a thread ofexecution, and/or the like. Still further, results of acts of themethodologies can be stored in a computer-readable medium, displayed ona display device, and/or the like.

Turning now to FIG. 8, an exemplary methodology 800 for outputting ananswer generated by a collective of crowd workers is illustrated. Themethodology 800 starts at 802, and at 804 a text string that comprises asequence of words is received. The text string can be from a query ormessage set forth by a user through utilization of a keyboard. Inanother example, the text string may be a transcription of a query ormessage audibly set forth by the user. At 806, the text string is parsedto identify that the text string represents an information need that isdesirably answered by a collective of crowd workers. For example, thetext string can be parsed to identify that the text string includes aquestion, and that the question is desirably answered by a collective ofcrowd workers.

At 808, responsive to the identifying that the information need isdesirably answered by the collective the crowd workers, content can betransmitted to a first plurality of computing devices operated by arespective first plurality of crowd workers from amongst the collectiveof crowd workers. The content includes first instructions to be followedby each crowd worker in the first plurality of crowd workers, whereinthe first instructions direct each crowd worker in the first pluralityof crowd workers to perform a same task. For instance, such task may beto generate a proposed answer for the information need.

At 810, responses from each crowd worker to the task are received, andat 812 a final answer to the information need is output based at leastin part upon the respective responses to the task from each crowd workerin the first plurality of crowd workers. The methodology 800 completesat 814.

Turning now to FIG. 9, an exemplary methodology 900 for outputting ananswer to a question submitted by way of a social networking applicationis illustrated. The methodology 900 starts at 902, and at 904, publicdata of a social networking application is monitored for questionsposted by users of the social networking application. At 906, a messageis identified as including a question that is desirably answered by acollective of crowd workers. Such message may be a status update or someother suitable message. At 908, the question and first instructions aretransmitted to a first plurality of computing devices operated by arespective first plurality of crowd workers. Other supplementalinformation may also be provided to the first plurality of crowdworkers.

At 910, proposed answers to the question are received from each crowdworker in the first plurality of crowd workers. In other words, everycrowd worker submits a proposed answer to the question. At 912, theproposed answers submitted by crowd workers in the first plurality ofcrowd workers are transmitted together with second instructions to asecond plurality of computing devices operated by a respective secondplurality of crowd workers. The second instructions instruct crowdworkers in the second plurality of crowd workers to vote on whichproposed answer is the best answer to the question. At 914, indicationsare received from the second plurality of crowd workers as to which ofthe answers is the best answer to the question from amongst the proposedanswers set forth by crowd workers in the first plurality of crowdworkers.

At 916, a final answer to the question is selected based at least inpart upon the indications received from the second plurality of crowdworkers at 914. For example, the answer receiving the most votes bycrowd workers in the second plurality of crowd workers can be selectedas the final answer. At 918, a final answer to the query is output as aportion of public data in the social networking application. Themethodology 900 completed 920.

Referring collectively to FIGS. 10 and 11, an exemplary methodology 1000for generating crowd-sourced answers to information needs identifiedthrough analysis of a search log is illustrated. The methodology 1000starts at 1002, and at 1004, search logs of a search engine are analyzedto identify a candidate web page. As noted above, a candidate web pageis a web page that is often a destination web page for queries that havecertain features (e.g., include question words). At 1006, the candidateweb page, queries corresponding thereto, and first instructions aretransmitted to a first plurality of computing devices operated by afirst plurality of crowd workers. The first instructions instruct crowdworkers in the first plurality of crowd workers to extract a portion ofthe candidate web page that a respective crowd worker believes to bestanswer the information need represented by the candidate web page andcorresponding queries. At 1008, portions of the candidate web pageextracted by respective workers in the first plurality of crowd workersare received from the first plurality of computing devices. While theabove refers to extracting portions of a candidate web page, it is to beunderstood that other resources may be provided, and content can beextracted from such resources in connection with providing acrowdsourced answer to an information need.

At 1010, the portions submitted by the first plurality of crowd workers,second instructions, and the corresponding queries are transmitted to asecond plurality of computing devices operated by a respective secondplurality of crowd workers. The second instructions instruct the secondplurality of crowd workers to vote on which portion best answers theinformation need represented by the queries. The second plurality ofcrowd workers operate the respective second plurality of computingdevices to submit their votes as to which portion of the candidate webpage best answers the information need.

At 1012, the votes submitted by the second plurality of crowd workersare received from the second plurality of computing devices. At 1014,the portion of the candidate web page receiving the highest number ofvotes and third instructions are transmitted to a third plurality ofcomputing devices operated by a respective third plurality of crowdworkers. The third instructions instruct the third plurality of crowdworkers to edit the portion of the candidate web page in accordance withdefined criteria (e.g., the portion is to be shortened to include nomore than fifteen words). The third plurality of crowd workers canutilize the respective third plurality of computing devices to submitedits.

At 1016, the edits submitted by the third plurality of crowd workers arereceived from the third plurality of computing devices. At 1018,responsive to receiving the edits, the edits and fourth instructions aretransmitted to a fourth plurality of computing devices operated by arespective fourth plurality of crowd workers. The fourth instructionsinstruct the fourth plurality of crowd workers to submit votes at towhich edit represents a best final answer to the information need. Thefourth plurality of crowd workers operate the fourth plurality ofcomputing devices to submit such votes.

At 1020, the votes are received from the fourth plurality of computingdevices, and at 1022, the edit receiving the highest number of votes isoutput as an approved crowd-sourced answer to the information need.Optionally, while not shown, other crowd workers can be instructed tosubmit a short title to the approved answer to assign a short title tothe crowd-sourced answer to the information need and such title can bevoted on by still other crowd workers.

With reference now to FIG. 12, an exemplary methodology 1200 foroutputting search results that include a crowd-sourced answer isillustrated. The methodology 1200 starts at 1202, and at 1204, a queryis received at a search engine. At 1206, a determination is made as towhether the query corresponds to a crowd-sourced answer. As noted above,the query can be analyzed to ascertain if it includes particularfeatures, and search results retrieved by the search engine can beanalyzed to ascertain if such search results include a web pageidentified as being a candidate web page.

If the query is not found to correspond to a crowd-sourced answer, thenat 1208, a conventional SERP is output to the issuer of the query. If at1206, however, it is found that the query corresponds to a crowd-sourcedanswer, then at 1210, a SERP is output that includes a crowd-sourcedanswer. In such case, the SERP can include conventional web page titlesand snippets as well as a crowd-sourced answer positioned inline withthe web page titles and snippets. The methodology 1200 completes at1212.

Now referring to FIG. 13, an exemplary methodology 1300 that facilitatesoutputting a crowd-sourced answer responsive to receipt of a query at asearch engine is illustrated. The methodology 1300 starts at 1302, andat 1304 a query is received at a search engine. At 1306, it isdetermined that the query represents an information need that isdesirably answered by a collective of crowd workers. This determinationcan be made based upon features corresponding to the query, adetermination that the query is time-critical in nature, etc.

At 1308, a search is executed by a search engine based upon the query.At 1310, search results (conventional) are output based upon the searchover the search index, wherein the search results comprise an indicationthat a collective of crowd workers is generating an answer to theinformation need of the user. At 1312, the query and first instructionsare transmitted to a first plurality of computing devices operated by arespective first plurality of crowd workers. Additionally, supplementalinformation can be transmitted to the first plurality of computingdevices, such as web pages that may include an answer to the informationneed of the user, information about the user, etc. The firstinstructions instruct the first plurality of crowd workers to submitproposed answers to the query.

At 1314, the proposed answers are received from computing devicesoperated by each crowd worker in the first plurality of crowd workers.At 1316, the proposed answers and second instructions are transmitted toa second plurality of computing devices operated by a respective secondplurality of crowd workers. The second instructions instruct crowdworkers in the second plurality of crowd workers to vote on whichproposed answer proffered by crowd workers in the first plurality ofcrowd workers best answers the information need of the user representedby the query.

At 1318, votes from the second plurality of crowd workers are receivedas to which proposed answer provided by crowd workers in the firstplurality of crowd workers best answers the information need. At 1320,search results are updated to include the answer voted by the secondplurality of crowd workers as best answering the information need. Themethodology 1300 completes 1322.

Referring now to FIG. 14, a high-level illustration of an exemplarycomputing device 1400 that can be used in accordance with the systemsand methodologies disclosed herein is illustrated. For instance, thecomputing device 1400 may be used in a system that generatescrowd-sourced answers responsive to a query being submitted to a searchengine. By way of another example, the computing device 1400 can be usedin a system that generates crowd-sourced answers to questions posted inmessages of a social networking application. The computing device 1400includes at least one processor 1402 that executes instructions that arestored in a memory 1404. The instructions may be, for instance,instructions for implementing functionality described as being carriedout by one or more components discussed above or instructions forimplementing one or more of the methods described above. The processor1402 may access the memory 1404 by way of a system bus 1406. In additionto storing executable instructions, the memory 1404 may also storesearch trails, proposed answers, instructions, and so forth.

The computing device 1400 additionally includes a data store 1408 thatis accessible by the processor 1402 by way of the system bus 1406. Thedata store 1408 may include executable instructions, instructions,proposed answers, votes, etc. The computing device 1400 also includes aninput interface 1410 that allows external devices to communicate withthe computing device 1400. For instance, the input interface 1410 may beused to receive instructions from an external computer device, from auser, etc. The computing device 1400 also includes an output interface1412 that interfaces the computing device 1400 with one or more externaldevices. For example, the computing device 1400 may display text,images, etc. by way of the output interface 1412.

It is contemplated that the external devices that communicate with thecomputing device 1400 via the input interface 1410 and the outputinterface 1412 can be included in an environment that providessubstantially any type of user interface with which a user can interact.Examples of user interface types include graphical user interfaces,natural user interfaces, and so forth. For instance, a graphical userinterface may accept input from a user employing input device(s) such asa keyboard, mouse, remote control, or the like and provide output on anoutput device such as a display. Further, a natural user interface mayenable a user to interact with the computing device 1400 in a mannerfree from constraints imposed by input device such as keyboards, mice,remote controls, and the like. Rather, a natural user interface can relyon speech recognition, touch and stylus recognition, gesture recognitionboth on screen and adjacent to the screen, air gestures, head and eyetracking, voice and speech, vision, touch, gestures, machineintelligence, and so forth.

Additionally, while illustrated as a single system, it is to beunderstood that the computing device 1400 may be a distributed system.Thus, for instance, several devices may be in communication by way of anetwork connection and may collectively perform tasks described as beingperformed by the computing device 1400.

Various functions described herein can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes computer-readable storage media. A computer-readablestorage media can be any available storage media that can be accessed bya computer. By way of example, and not limitation, suchcomputer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to carry or storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. Disk and disc, as used herein,include compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk, and blu-ray disc (BD), where disks usuallyreproduce data magnetically and discs usually reproduce data opticallywith lasers. Further, a propagated signal is not included within thescope of computer-readable storage media. Computer-readable media alsoincludes communication media including any medium that facilitatestransfer of a computer program from one place to another. A connection,for instance, can be a communication medium. For example, if thesoftware is transmitted from a website, server, or other remote sourceusing a coaxial cable, fiber optic cable, twisted pair, digitalsubscriber line (DSL), or wireless technologies such as infrared, radio,and microwave, then the coaxial cable, fiber optic cable, twisted pair,DSL, or wireless technologies such as infrared, radio and microwave areincluded in the definition of communication medium. Combinations of theabove should also be included within the scope of computer-readablemedia.

Alternatively, or in addition, the functionally described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Program-specific Integrated Circuits (ASICs), Program-specificStandard Products (ASSPs), System-on-a-chip systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), etc.

What has been described above includes examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable modification and alteration of the above devices ormethodologies for purposes of describing the aforementioned aspects, butone of ordinary skill in the art can recognize that many furthermodifications and permutations of various aspects are possible.Accordingly, the described aspects are intended to embrace all suchalterations, modifications, and variations that fall within the spiritand scope of the appended claims. Furthermore, to the extent that theterm “includes” is used in either the details description or the claims,such term is intended to be inclusive in a manner similar to the term“comprising” as “comprising” is interpreted when employed as atransitional word in a claim.

What is claimed is:
 1. A method, comprising: receiving, at a computingdevice, a text string that comprises a sequence of words; parsing thetext string; based at least in part upon the parsing of the text string,identifying that the text string represents an information need that isdesirably answered by a collective of crowd workers; responsive to theidentifying that the text string represents an information need that isdesirably answered by the collective of crowd workers, transmittingcontent to a first plurality of computing devices operated by arespective first plurality of crowd workers from amongst the collectiveof crowd workers, the content comprising first instructions thatinstruct each crowd worker in the first plurality of crowd workers toperform a same task; receiving respective responses to the task fromeach crowd worker in the first plurality of crowd workers; andoutputting an answer to the information need based upon the responses tothe task from the first plurality of crowd workers.
 2. The method ofclaim of claim 1, wherein the text string comprises a query submitted toa search engine, the method further comprising: responsive toidentifying that the query represents the information need that isdesirably answered by the collective of crowd workers, identifying atleast one resource related to the information need; and transmitting theat least one resource to each crowd worker in the first plurality ofcrowd workers, wherein the first instructions instruct each crowd workerin the first plurality of crowd workers to generate a proposed answerbased upon the query and the at least one resource.
 3. The method ofclaim 2, wherein the first instructions instruct each crowd worker inthe first plurality of crowd workers to generate the proposed answer byextracting content from the at least one resource.
 4. The method ofclaim 3, wherein the at least one resource is a resource retrieved byexecuting a search over a search engine index based upon the query, themethod further comprising: receiving portions of the resource extractedtherefrom by respective crowd workers in the first plurality of crowdworkers; transmitting the portions, the query, and second instructionsto a second plurality of computing devices operated by a secondplurality workers, the second instructions requesting each crowd workerin the second plurality of crowd workers to set forth an indication of apreference as to which option of several options best answers theinformation need, the several options comprising the portions of thedocument extracted therefrom by respective crowd workers in the firstplurality of crowd workers; receiving indications of preferences fromthe second plurality of computing devices; and outputting the answerbased at least in part upon the indications of the preferences.
 5. Themethod of claim 1, wherein the text string is received from a publicfeed of a social networking application, and wherein the text stringcomprises an explicit or implied question.
 6. The method of claim 5,wherein the task is for each crowd worker from the first plurality ofcrowd workers to submit a respective proposed answer to the question. 7.The method of claim 6, further comprising: responsive to receivingproposed answers to the question submitted by the first plurality ofcrowd workers, transmitting the proposed answers and second instructionsto a second plurality of computing devices operated by a respectivesecond plurality of crowd workers, the second instructions instructingeach crowd worker in the second plurality of crowd workers to submit arespective vote as to which of several options best answers theinformation need, the several options comprising the proposed answers;receiving votes submitted by the second plurality of crowd workers,wherein the answer is output based at least in part upon the votes. 8.The method of claim 1, wherein the identifying that the text stringrepresents an information need that is desirably answered by thecollective of crowd workers comprises: determining that the text stringcomprises at least one predefined feature; responsive to determiningthat the text string comprises the at least one predefined feature,transmitting at least a portion of the text string and secondinstructions to a computing device of a crowd worker from the collectiveof crowd workers, the second instructions instructing the crowd workerto indicate whether or not the information need is answerable by thecollective of crowd workers; and receiving a response from the crowdworker that the information need is answerable by the collective ofcrowd workers.
 9. The method of claim 1, wherein identifying that thetext string represents an information need that is desirably answered bythe collective of crowd workers comprises identifying that the textstring comprises a question or query that is time-critical in nature.10. The method of claim 1, wherein the outputting of the answer to theinformation need comprises including the answer to the information needin a search engine results page, the search engine results pagecomprising a ranked list of URLs, and wherein the answer is presented inthe search engine results page to visually differentiate the answer fromURLs in the ranked list of URLs.
 11. The method of claim 10, wherein theanswer is displayed above the URLs in the ranked list of URLs.
 12. Asystem, comprising: a processor; and a memory that comprises a pluralityof components that are executed by the processor, the plurality ofcomponents comprising: a classifier component that receives a textstring and classifies the text string as representing an informationneed that is desirably answered by a collective of crowd workers; aninstruction transmitter component that transmits the text string andfirst instructions to a first plurality of computing devices operated bya first plurality of crowd workers in the collective of crowd workers,the first instructions instructing each crowd worker in the firstplurality of crowd workers to perform a same task with respect to theinformation need; a response receiver component that receives responsesto the task submitted by the first plurality of crowd workers; and anoutput component that outputs an answer to the information need based atleast in part upon the responses to the task received by the responsereceiver component.
 13. The system of claim 12 being comprised by asearch engine, wherein the text string comprises a query submitted tothe search engine.
 14. The system of claim 12, wherein the text stringcomprises a question included in a message posted to a feed of a socialnetworking application, the system further comprising an analyzercomponent that analyzes the feed and identifies that the messageincludes the question, wherein the classifier receives the questionresponsive to the analyzer component identifying that the messageincludes the question.
 15. The system of claim 12, wherein theclassifier component classifies the text string as representing aninformation need that is desirably answered by the collective of crowdworkers based at least in part upon a non-alphanumerical character beingincluded in the text string.
 16. The system of claim 12, wherein textstring comprises a question, and wherein the task is to submit aproposed answer to the question.
 17. The system of claim 16, whereinresponsive to the response receiver component receiving proposed answersto the question, the instruction transmitter component transmits thequestion, the proposed answers, and second instructions to a secondplurality of computing devices operated by a second plurality of crowdworkers, the second instructions instructing each crowd worker in thesecond plurality of crowd workers to submit a vote as to which proposedanswer best answers the information need, and wherein the outputcomponent outputs the answer to the information need based at least inpart upon votes submitted by the second plurality of crowd workers. 18.The system of claim 12, further comprising an accounting component thattracks payment to be provided to each crowd worker in the firstplurality of crowd workers for their respective responses.
 19. Thesystem of claim 12, wherein the output component outputs the answer to apublic feed of a social networking application, the answer displayed inthe public feed in correspondence with a message that includes the textstring.
 20. A computer-readable storage medium comprising instructionsthat, when executed by a processor, cause the processor to perform actscomprising: monitoring a public feed of a social networking applicationfor questions included in messages posted to the public feed;determining that a message posted to the public feed by a user of thesocial networking application comprises a question; determining that themessage posted to the public feed by the user of the social networkingapplication comprises a hash tag; identifying that the question isdesirably answered by a collective of crowd workers based upon thedetermining that the message comprises the question and the determiningthat the message comprises the hash tag; transmitting the question andfirst instructions to a first plurality of computing devices operated bya respective first plurality of crowd workers, the first instructionsinstructing each crowd worker in the first plurality of crowd workers togenerate a respective proposed answer to the question; receivingproposed answers to the question submitted by the first plurality ofcrowd workers; responsive to receiving the proposed answers to thequestion, transmitting the proposed answers to the question and secondinstructions to a second plurality of computing devices operated by arespective second plurality of crowd workers, the second instructionsinstructing each crowd worker in the second plurality of crowd workersto submit a vote as to which of the proposed answers best answers thequestion; receiving votes submitted by the second plurality of crowdworkers; selecting a final answer to the question based at least in partupon the votes; and outputting the final answer to the question to thepublic feed.